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(57) ABSTRACT 

A method for generating color video signals representative of 
color images of a scene includes the following steps: focusing 
light from the scene on an electronic image sensor via a filter 
having a tri-color filter pattern; producing, from outputs of the 
sensor, first and second relatively low resolution luminance 
signals; producing, from outputs of the sensor, a relatively 
high resolution luminance signal; producing, from a ratio of 
the relatively high resolution luminance signal to the first 
relatively low resolution luminance signal, a high band lumi- 
nance component signal; producing, from outputs of the sen- 
sor, relatively low resolution color component signals; and 
combining each of the relatively low resolution color compo- 
nent signals with the high band luminance component signal 
to obtain relatively high resolution color component signals. 

24 Claims, 4 Drawing Sheets 


g 2 - 


306 ^ 

Convert G 
to Y Using 
Adjacent 
B and R 

~~~T~ 


4 - 


,307 


Convert G 
to Y Using 
Adjacent 
B and R 


*2 


-C 


Gain 
Correct 
to Match 

Yl 


Filter 

and 

Scan Convert 


Y L 1 


340 - 



Filter 

.I 


i „ 

ond 

Log 7 


Scan 



n 

Convert 




325 


Log Y H 

+ -Log (y h /y l ) 


Filter 

Y|_ H 


and 

Log 7 

Scan 


Convert 

326 

7"^ 


Log Y L 




362 


Log G|_ 


Color 

Matrix 


RL^^ LogRu ^^ Lo9 Rh 


350 




J5 

366 


F=- HLog y\ - 


372 




374 
— ► I 
376 


Log G H 


Log B L Bh 








US 7,688,362 B2 

Page 2 


U.S. PATENT DOCUMENTS 


6,570,616 Bl* 5/2003 Chen 348/272 

6,614,471 Bl* 9/2003 Ott 348/238 

6,778,220 B2 8/2004 Glenn 348/368 

6,847,397 Bl * 1/2005 Osada 348/273 

6,882,364 Bl* 4/2005 Inuiyaetal 348/252 

6,891,568 Bl 5/2005 Glenn 348/262 


7,082,218 B2 * 7/2006 Pollard et al 382/167 

7,230,646 B2* 6/2007 Glenn 348/315 


FOREIGN PATENT DOCUMENTS 

JP 07-038915 * 2/1995 

JP 07038915 A * 2/1995 

* cited by examiner 



U.S. Patent 


Mar. 30, 2010 


Sheet 1 of 4 


US 7,688,362 B2 


. t 

R 

G 

R 

G 

R 

— 

G 

B 

G 

B 

G 

— 

R 

G 

R 

G 

R 

— 

G 

B 

G 

B 

G 



R 

G 

R 

G 

R 

— 

G 

B 

G 

B 

G 

— 

l 

i 

\ 

t 

1 



i i t i 


PRIOR ART 


U.S. Patent 


Mar. 30, 2010 


Sheet 2 of 4 


210 



A 



260 


P fi ^ 

PIG. 



2 


US 7,688,362 B2 


245 



270 


























































US 7,688,362 B2 


1 

SINGLE SENSOR PROCESSING TO OBTAIN 
HIGH RESOLUTION COLOR COMPONENT 
SIGNALS 

RELATED APPLICATION 5 

The present application claims priority from U.S. Provi- 
sional Patent Application Ser. 60/551,521, filed Mar. 5, 2004, 
and said Provisional Patent Application is incorporated 
herein by reference. 10 

GOVERNMENT RIGHTS 

This invention was made with Government support under 
NASA Contract NCC8-272. The Government has certain 15 
rights in the invention. 

FIELD OF THE INVENTION 

This invention relates to color video camera processing 20 
and, more particularly, to a method and apparatus for obtain- 
ing improved color video images from single sensor video 
cameras that use a tricolor filter. 

BACKGROUND OF THE INVENTION 25 

Single sensor color video cameras typically utilize a color 
filter to separate the color components of an image. The pixels 
or pixel regions on the sensor (for example a CCD or CMOS 
chip) are read out, or scanned out, to produce video color 30 
component signals from the pixels or pixel regions associated 
with respective colors, for example, green, red, and blue 
stripe regions or checkerboard regions on the sensor that 
receive the corresponding filtered light of the image. In this 
manner, color separation can be achieved. 35 

One type of filter, which has advantages for single sensor 
operation is a so-called Bayer pattern filter, illustrated in FIG. 

1 . This filter has a checkerboard type of pattern with green 
filter regions arranged diagonally as every other square on 
every line. The red and blue filter regions alternate on the 40 
diagonals and are present as every other square on every other 
line. In another possible arrangement, the green squares are 
instead white, with green being derived from luminance and 
the other colors. 

As seen in FIG. 1, only half of the total filter regions pass 45 
green, only one quarter of the total filter regions pass red, and 
only one quarter of the total filter regions pass blue. As in 
other single sensor color systems, there are trade-offs 
between the need to separate colors, and the desire to also 
achieve reasonable good resolution in color, notwithstanding 50 
the fact that, after filtering, only a fraction of the light of each 
color is actually incident on the sensor surface. 

It is among the objects of the present invention to improve 
performance of single sensor video cameras by processing to 
enhance perceived color resolution. 55 

SUMMARY OF THE INVENTION 

In embodiments of the present invention, isoluminant color 
is used to get lull resolution RGB for saturated colors, and 60 
improved resolution for changes in color. In embodiments 
hereof, the processing uses to advantage the fact that the green 
primary has so much red and blue cross talk, that it is close to 
photopic luminance. 

In accordance with an embodiment of the method of the 65 
invention, there is disclosed a method for generating color 
video signals representative of color images of a scene, 


2 

including the following steps: focusing light from the scene 
on an electronic image sensor via a filter having a tri -color 
filter pattern; producing, from outputs of the sensor, first and 
second relatively low resolution luminance signals; produc- 
ing, from outputs of the sensor, a relatively high resolution 
luminance signal; producing, from a ratio of the relatively 
high resolution luminance signal to the first relatively low 
resolution luminance signal, a high band luminance compo- 
nent signal; producing, from outputs of the sensor, relatively 
low resolution color component signals; and combining each 
of the relatively low resolution color component signals with 
the high band luminance component signal to obtain rela- 
tively high resolution color component signals. 

In an illustrated embodiment of the invention, the filter has 
a green-red-blue Bayer filter pattern. 

In a preferred embodiment of the invention, the step of 
producing, from outputs of the sensor, first and second rela- 
tively low resolution luminance signals, includes producing 
first and second green color signals from outputs of the sen- 
sor, and modifying the first and second green color signals 
with adjacent red and blue signals from outputs of the sensor, 
to produce the first and second low resolution luminance 
signals. Also in this embodiment, the first and second low 
resolution luminance signals comprise luminance signals 
derived respectively from green pixels on odd lines of outputs 
of the sensor and from green pixels on even lines of outputs of 
the sensor. Also in this embodiment, The step of producing, 
from outputs of the sensor, a relatively high resolution lumi- 
nance signal, comprises combining the first and second low 
resolution luminance signals and producing a scan-converted 
relatively high resolution luminance signal. Also in this 
embodiment, the step of producing, from outputs of the sen- 
sor, relatively low resolution color component signals, 
includes producing from outputs of the sensor, and the first 
relatively low resolution luminance signal, scan converted 
relatively low resolution color component signals . Also in this 
embodiment, the step of combining each of the relatively low 
resolution color component signals with the high band lumi- 
nance component signal to obtain relatively high resolution 
color component signals includes combining each of the scan 
converted relatively low resolution color component signals 
with the high band luminance component signal. 

In another preferred embodiment of the invention the high 
band luminance component signal is of the form (Y^Afy), 
and step of combining each of the scan converted relatively 
low resolution color component signals with the high band 
luminance component signal to obtain relatively high resolu- 
tion green, red, and blue color component signals comprises 
deriving signals of the form log G (Y // /Y i ), log R (Y i? /Y i ), 
andlogBQVYJ. 

Further features and advantages of the invention will 
become more readily apparent from the following detailed 
description when taken in conjunction with the accompany- 
ing drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. lisa diagram of a Bayer pattern color filter that can 
be employed in embodiments of the invention. 

FIG. 2 is block diagram of a camera system in accordance 
with an embodiment of the invention, and which can be used 
in practicing the method of the invention. 

FIG. 3 is a block diagram of the processing circuitry of 
FIG. 2 in accordance with an embodiment of the invention, 
and which can be used in practicing the method of the inven- 
tion. 
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FIG. 4 is a block diagram of the processing circuitry of 
FIG. 2 in accordance with another embodiment of the inven- 
tion, and which can be used in practicing the method of the 
invention. 

DETAILED DESCRIPTION 

Referring to FIG. 2, there is shown a block diagram of a 
video camera system 200 in accordance with an embodiment 
of the apparatus of the invention and which can be used in 
practicing an embodiment of the method of the invention. 
Light received from a scene 210 being viewed (i.e., a scene 
which can be moving) is focused by a camera lens system 220 
through a color filter 230 onto an image sensor 240. In the 
present embodiment, the color filter 230 is a Bayer pattern 
filter, as in FIG. 1, and the image sensor may be, for example, 
a 3840x2160 CMOS of a type used for high definition video 
applications, although it will be understood that the principles 
of the invention are applicable to any suitable type of image 
sensor. The sensor output is scanned by circuit 245, which 
produces as outputs, the raw color signals 245A which are 
coupled with processing circuitry 250 in accordance with 
embodiments of the invention, and which is shown in further 
detail in FIGS. 3 and 4. The processed video signals can 
ultimately be displayed on any suitable display 260, such as a 
high definition television display 260, and/or recorded on 
recorder 270. 

FIG. 3 is a block diagram of an embodiment of the pro- 
cessing circuitry 250 of FIG. 2, the green color signal corre- 
sponding to every other pixel on the odd lines of the Bayer 
pattern, and designated G x , is coupled to block 306, and the 
green color signal corresponding to every other pixel on the 
even lines of the Bayer pattern, and designated G 2 , is coupled 
to block 307. Both of these are at 1920x1080 pixels in the 
present example. Each of these blocks 306, 307 also receives 
the red color and blue color signals, and operates to convert 
green (G) to luminance (Y) by combining appropriate frac- 
tions of adjacent red (R) and blue (B) for each pixel. The 
results are designated Y x andY 2 . The Y 2 signal is gain cor- 
rected to correct for crosstalk, as represented by the block 
311, andY : andY 2 are coupled to block321 which operates to 
filter, combine, and interpolate to obtain a scan-converted 
3840x2160 luminance, which is relatively high resolution 
luminance signal designated Y H . One of the luminance com- 
ponents (Y x in this example) is also filtered and interpolated, 
as represented by block 325, to obtain a scan-converted 
3840x2160 luminance, which is a relatively low resolution 
luminance signal with an increased number of pixels, and 
designated Y L . In the present embodiment, log gamma cir- 
cuits 322 and 326 are used to derive log Y^ and log Y z , and 
these logs are subtracted, as represented by block 330, to 
obtain the signal log (YjJY l ). 

The signal Y 1 is also one input to the block 340, which also 
receives, via aligning delays 339, the red color signal corre- 
sponding to every other pixel on the odd lines and the blue 
color signal corresponding to every other pixel on the even 
lines. In the present example, these red and blue signals are at 
1 920x1080 pixels. The scan conversion of block 340, which, 
as in all scan conversions hereof, can employ interpolation 
with any desired number of taps, increases the number of 
pixels of the signals to 3840x2160. It will be understood that 
the scan conversion can include any necessary position 
adjustment. The outputs of block 340, designatedY zl , R, and 
B, are input to color matrix 350 which is operative to convert 
the relatively low resolution Y to green by subtracting out 
appropriate portions of the red and blue components. The 
outputs of matrix 350 are relatively low resolution color com- 
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ponent signals (at a relatively low resolution, but with pixel 
count increased to 3840x2160), designated G L , R L and B L , 
and are respectively coupled to log gamma circuits 362, 364, 
and 366. The outputs of the log gamma circuits, respectively 
5 designated as log G L , log R L , and log B L , are added, by 
summing circuits 372, 374 and 376, to the high resolution 
component log (Yj/Y l ), to obtain gamma corrected rela- 
tively high resolution color components Rjy, and B^^. In 
the present example, the output color component signals are 
to at a resolution of 3840x2 1 60 for saturated colors and 1 920x 
1080 for changes in color. It can be noted that, if desired, the 
implemented ratios (difference of logs) and products (sums of 
logs) can be achieved using, for example, digital ratios and 
products (such as can be implemented using field program- 
1 5 mable gate arrays) to generate outputs that can then be gamma 
corrected. However, one of the advantages of the illustrated 
processing is that the output component signals are already 
gamma correlated. 

FIG. 4 is a block diagram of a further embodiment which 
20 implements processing sequentially, rather than simulta- 
neously as in FIG. 3. In the example of FIG. 4, the output of 
block 245 of FIG. 2 comprises interlaced fields from the 
sensor used in conjunction with the Bayer pattern color filter. 
In FIG. 4, the signals from the green pixels (that is, the sensor 
25 regions corresponding to the green filter areas) are coupled, 
via a one pixel delay 402, to a block 405, which is similar to 
the blocks 306 and 307 of FIG. 3. In particular, the block 405 
operates to correct G to Y, based on the adjacent R and B, and 
implements any necessary gain correction, in this case at the 
30 field rate. The output is designated Y^. The signal Y H , and the 
one field delayed version thereof (from block 407) are 
summed by summing circuit 408, and then input to demulti- 
plexer 410, which receives the video timing signals and pro- 
duces the separated relating lower resolution odd field, Y L1 , 
35 and even field, Y i2 . 

The signals representing the red and blue pixels of the 
Bayer pattern are input to a two pixel delay 412 and the output 
thereof is summed with the input signal by a summing circuit 
414, and the sum is demultiplexed by demultiplexer 416, to 
40 produce odd field red and even field blue signals. The two 
pixel delay averaging aligns the red and blue pixel signals 
with the green. The outputs of demultiplexer 416 are respec- 
tively coupled to circuits that include a one field delay (blocks 
417 and 418) and a summing circuit (blocks 421 and 422). 
45 These circuits are operative to repeat the field so that the red 
signal, designated R L and the blue signal, designated B L , are 
present during both fields. 

Referring next to the lower portion of the diagram, a 
divider circuit 431 receives the signal Y^ and the signal Y z for 
50 the odd lines and produces Y H (Y L for the odd lines, which is 
multiplied (multiplier circuit 436) by the red signal R L . The 
result is a high resolution red color signal R^, equal to R L 
(Yj/Yj). Similarly, a divider circuit 432 receives the signal 
Y h and the signal Y L for the even lines and produces Y H /Y L 
55 for the even lines, which is multiplied (multiplier circuit 438) 
by the red signal R L . The result is a high resolution red color 
signal B^, equal to B L (Yj/Y L ). A color matrix 450 operates 
in similar manner to the color matrix 350 of the FIG. 3 
embodiment; namely, it receives Y^, R^ and B^ and operates 
60 to subtract out the red and blue components from Y* to 
produce the high band green color signal, G H . 

The invention has been described with reference to particu- 
lar preferred embodiments, but variations within the spirit 
and scope of the invention will occur to those skilled in the art. 
65 For example, if the image sensor is upgraded to operate at a 
higher frame rate, the exemplary embodiments will produce 
signals at proportionally higher rates. Also, while green, red 
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and blue, are preferred filter colors, it will be understood that 
other combinations, for example subtractive colors, could be 
used. 

The invention claimed is: 

1. A method for generating color video signals representa- 
tive of color images of a scene, comprising the steps of: 

focusing light from said scene on an electronic image sen- 
sor via a filter having a tri -color filter pattern; 

producing, from outputs of said sensor, first and second 
relatively low resolution luminance signals; 

producing, from outputs of said sensor, a relatively high 
resolution luminance signal, by interpolating said first 
and second low resolution luminance signals to obtain a 
scan-converted relatively high resolution luminance sig- 
nal; 

producing, from a ratio of said relatively high resolution 
luminance signal to said first relatively low resolution 
luminance signal, a high band luminance component 
signal; 

producing, from outputs of said sensor, relatively low reso- 
lution color component signals; and 

combining each of said relatively low resolution color 
component signals with said high band luminance com- 
ponent signal to obtain relatively high resolution color 
component signals. 

2. The method as defined by claim 1, wherein said step of 
focusing light from said scene on an electronic image sensor 
via a filter having a tri -color filter pattern comprises focusing 
light from said scene on an electronic image sensor via a filter 
having a green-red-blue filter pattern. 

3. The method as defined by claim 2, wherein said step of 
producing, from outputs of said sensor, first and second rela- 
tively low resolution luminance signals, includes producing 
first and second green color signals from outputs of said 
sensor, and modifying said first and second green color sig- 
nals with adjacent red and blue signals from outputs of said 
sensor, to produce said first and second low resolution lumi- 
nance signals. 

4. The method as defined by claim 3, wherein said first and 
second low resolution luminance signals comprise luminance 
signals derived respectively from green pixels on odd lines of 
outputs of said sensor and from green pixels on even lines of 
outputs of said sensor. 

5. The method as defined by claim 2, wherein said first and 
second low resolution luminance signals comprise luminance 
signals derived respectively from green pixels on odd lines of 
outputs of said sensor and from green pixels on even lines of 
outputs of said sensor. 

6. The method as defined by claim 2, wherein said step of 
producing, from outputs of said sensor, relatively low reso- 
lution color component signals, includes producing from out- 
puts of said sensor, and said first relatively low resolution 
luminance signal, scan converted relatively low resolution 
color component signals. 

7. The method as defined by claim 1, wherein said step of 
focusing light from said scene on an electronic image sensor 
via a filter having a tri -color filter pattern comprises focusing 
light from said scene on an electronic image sensor via a filter 
having a green-red-blue Bayer filter pattern. 

8. The method as defined by claim 7, wherein said step of 
producing, from outputs of said sensor, first and second rela- 
tively low resolution luminance signals, includes producing 
first and second green color signals from outputs of said 
sensor, and modifying said first and second green color sig- 
nals with adjacent red and blue signals from outputs of said 
sensor, to produce said first and second low resolution lumi- 
nance signals. 
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9. The method as defined by claim 8, wherein said first and 
second low resolution luminance signals comprise luminance 
signals derived respectively from green pixels on odd lines of 
outputs of said sensor and from green pixels on even lines of 

5 outputs of said sensor. 

10. The method as defined by claim 8, wherein said step of 
producing, from outputs of said sensor, relatively low reso- 
lution color component signals, includes producing from out- 
puts of said sensor, and said first relatively low resolution 

to luminance signal, scan converted relatively low resolution 
color component signals. 

11. The method as defined by claim 7, wherein said first 
and second low resolution luminance signals comprise lumi- 
nance signals derived respectively from green pixels on odd 

15 lines of outputs of said sensor and from green pixels on even 
lines of outputs of said sensor. 

12. The method as defined by claim 1, wherein said step of 
focusing light from said scene on an electronic image sensor 
via a filter having a tri -color filter pattern comprises focusing 

20 light from said scene on an electronic image sensor via a filter 
having a white-red-blue Bayer filter pattern. 

13. The method as defined by claim 1, wherein said step of 
focusing light from said scene on an electronic image sensor 
via a filter having a tri -color filter pattern comprises focusing 

25 light from said scene on an electronic image sensor via a filter 
having a white-red-blue Bayer filter pattern. 

14. A method for generating color video signals represen- 
tative of color images of a scene, comprising the steps of: 

focusing light from said scene on an electronic image sen- 

30 sor via a filter having a tri-color filter pattern of green, 
red, and blue; 

producing, from green pixel outputs of said sensor, first and 
second relatively low resolution luminance signals; 

producing, from a combination of said first and second low 

35 resolution luminance signals, a scan-converted rela- 

tively high resolution luminance signal; 

producing, from a ratio of said relatively high resolution 
luminance signal to said first relatively low resolution 
luminance signal, a high band luminance component 

40 signal; 

producing, from outputs of said sensor and said first rela- 
tively low resolution luminance signal, scan converted 
relatively low resolution green, red and blue color com- 
ponent signals; and 

45 combining each of said scan converted relatively low reso- 
lution color component signals with said high band 
luminance component signal to obtain relatively high 
resolution green, red, and blue color component signals. 

15. The method as defined by claim 14, wherein said step of 

50 focusing light from said scene on an electronic image sensor 

via a filter having a tri-color filter pattern of green, red, and 
blue, comprises focusing light from said scene on an elec- 
tronic image sensor via a filter having a Bayer filter pattern. 

16. The method as defined by claim 15, wherein said step of 

55 producing, from green pixel outputs of said sensor, first and 

second relatively low resolution luminance signals, includes 
producing first and second green color signals from outputs of 
said sensor, and modifying said first and second green color 
signals with adjacent red and blue signals from outputs of said 

60 sensor, to produce said first and second low resolution lumi- 
nance signals. 

17. The method as defined by claim 15, wherein said high 
band luminance component signal is of the form (Y / /Y jL ), 
and wherein said step of combining each of said scan con- 

65 verted relatively low resolution color component signals with 
said high band luminance component signal to obtain rela- 
tively high resolution green, red, and blue color component 
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signals comprises deriving signals of the form log(G(Y jf / 
Y,)), logCRCY^)), and log^Y^)). 

18 . The method as defined by claim 15, wherein said step of 
producing, from a combination of said first and second low 
resolution luminance signals, a scan-converted relatively 
high resolution luminance signal, includes increasing the 
number of pixels of each of said first and second low resolu- 
tion luminance signals, and wherein said step of producing, 
from outputs of said sensor and said first relatively low reso- 
lution luminance signal, scan converted relatively low reso- 
lution green, red and blue color component signals, includes 
increasing the number of pixels of each of said relatively low 
resolution color component signals. 

19. The method as defined by claim 14, wherein said high 
band luminance component signal is of the form (YH/Y Z ), 
and wherein said step of combining each of said scan con- 
verted relatively low resolution color component signals with 
said high band luminance component signal to obtain rela- 
tively high resolution green, red, and blue color component 
signals comprises deriving signals of the form log(G(Y / / 

YJ), logCROVYJ), and log(H(Y„/YJ)- 

20 . The method as defined by claim 1 4 , wherein said step of 
producing, from a combination of said first and second low 
resolution luminance signals, a scan-converted relatively 
high resolution luminance signal, includes increasing the 
number of pixels of each of said first and second low resolu- 
tion luminance signals, and wherein said step of producing, 
from outputs of said sensor and said first relatively low reso- 
lution luminance signal, scan converted relatively low reso- 
lution green, red and blue color component signals, includes 
increasing the number of pixels of each of said relatively low 
resolution color component signals. 

21 . Apparatus for generating color video signals represen- 
tative of color images of a scene, comprising: 

means for focusing light from said scene on an electronic 
image sensor via a filter having a tri -color filter pattern; 

means for producing, from outputs of said sensor, first and 
second relatively low resolution luminance signals; 
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means for producing, from outputs of said sensor, a rela- 
tively high resolution luminance signal, comprising 
means for interpolating said first and second low reso- 
lution luminance signals to obtain a scan-converted rela- 

5 tively high resolution luminance signal; 

means for producing, from a ratio of said relatively high 
resolution luminance signal to said first relatively low 
resolution luminance signal, a high band luminance 
component signal; 

to means for producing, from outputs of said sensor, rela- 
tively low resolution color component signals; and 
means for combining each of said relatively low resolution 
color component signals with said high band luminance 
component signal to obtain relatively high resolution 

15 color component signals. 

22. Apparatus as defined by claim 21 wherein said means 
for of focusing light from said scene on an electronic image 
sensor via a filter having a tri-color filter pattern comprises 
means for focusing light from said scene on an electronic 

20 image sensor via a filter having a green-red-blue Bayer filter 
pattern. 

23. Apparatus as defined by claim 22, wherein said means 
for producing, from outputs of said sensor, first and second 
relatively low resolution luminance signals, includes means 

25 for producing first and second green color signals from out- 
puts of said sensor, and means for modifying said first and 
second green color signals with adjacent red and blue signals 
from outputs of said sensor, to produce said first and second 
low resolution luminance signals. 

30 24. Apparatus as defined by claim 23, wherein said means 

for producing, from outputs of said sensor, first and second 
low resolution luminance signals, comprises means for pro- 
ducing said first low resolution luminance signal from green 
pixels on odd lines of outputs of said sensors and producing 

35 said second low resolution luminance signal from green pix- 
els on even lines of outputs of said sensors. 



